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CHAPTER I 
INTRODUCTION 

The purpoia of thli guldt li to prcvld* atilitanc* in titabXlihlat 
the velldlty of the TU program et the ichool level. The procedures 
listed herein srs ususlly rsfsrrsd to ss product svslustion, thst is, 
evaluating ine outcones of the TU program. Most of the evaluation 
procedures can be conducted by a school counselor, an administrator, or 
someone else with a minimum of training in measurement and stattstlet. 
However, other aspects of the evaluation (particularly the analysis) 
require someone with special training. If ^the procedures in this gulda 
are followed and the TU program is successful at your school, this 
success can be demonstrated. . ' ^ 

Tho remainder of this guide is divided into two major sections. The 
first describes the procedures ^ich need to be followed to evaluate the 
program. The second illustrates how the results might be reported. *ithia 
guide also indicates the information needed J^the Talents Unlimited (TU) 
office in Mobile, Alabama for its evaluation as a Demonstrator /Developor* 
The forms to be used for this purpose are found in Appendices B and C. 
Your c^^eration in this respect is Appreciated. 



CHAPTER II 
PROCEDURES 

The procedural outilnrJ here ehould provide the framework for 
establishing the validity of a Talenta Unlimited Program at the achool 
level. The procedurea Include providing meaaurable objectlvea, aettlng 
up a cauae-effect dealgn, ualng valid and reliable meaaurlng InatruMOta 
and analyilng the data properly. Thla chapter deala with each of those 
concerns. 

^ Objectives 

In order for outcone objectives to function properly, they nuat be 
measurable and the efficiency of the TU process must be established by 
their attainment. That is, successful acconpllshment of the objectives 
should establish that the program la effective. Thus, the objectives 
become the basis for the evaluation design. 

Consider the following TU program objective! 

By the end of the school year, students receiving TU 
Instruction In the Productive Thinking talent will outperform 
aignifdcantly (at the .05 level) students not receiving TU 
instruction in this talent «a indicated by the Pr^^uetive 
Thinking Criterion Referenced laltnt Test. 

• ■ • * 

Basically, the objective indicates who (TU students) will do what 
(outperform non -TU students) when (by the end of the year) , how well 
(significantly), and what the meaaure la (Productive Thinking CRT). A 
complete aet of sample TU program objeotlvcfs is provided in Appendix A. 



Petlgn 

E.itabllihlng btyond a raatonablt doubt that the TU trtatnent caiiitd 
improvisd talant aecompllilyBtnti in itudenti requlrti tha uae of a TU 
trcatnuint ^roup and a similar comparlion group which did net receivt eha 
TU cre.itment. Thli lection providai raconiBtndations on how to let up 
this dualgn and tha procadurti outlinad ara intandad to ba appropriata for 
thu objectivaa atatad in tha pravioua aaction. 

Tlie noat affactivo method of setting up a treatment group/romparison 
group design la to aaaign atudenta in a random faahion to the two groupe. 
This can be acconpliahed by the following steps. First, aaaign numbers 
sequentially to an alphabetised list of potential atudent partidpanta* 
Then, write correaponding numbera on email alipa of paper and place tham 
In a hat or box. After thoroughly ahuf fling the numbera, draw out one- 
half of them. The atudenta whoae pumbera correspond to those drawn will 
be the TU treatment group and the balance will be the compariaon group. 
Teachera should alao be aailgned on a random baaia. The reaulting design 
would he aa follows i < 



Group 



Beginning 
of year 



Experimental 

Compariaon 



Randoji aelection 
Random aelection 



During year 

TU treatment 
Regular treatment 



End of 
year 

Poattest 
Poatteat 



In the event that the above procedure la not feaaible, the following 
compromlaea may be made. If a number of intact classrooms are available 
tor receiving the TU treatment, aequentlally number these elaases and 
itiloct one-half of tham in a random manner aa Indicated pravluusly* 
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These will be tht TU trt«tMiit elatift and tht rtmalnlng will bt tht 
comparUon claiita. It li bait to aalaot randmily both tha TU and 
comparlaon taachara fra« thoaa voluntaarlni Inataad of ualng TU voluntaara 
In the TU claaaaa whlla ualng non-voluntaara In tha conpailaon olaaaaa. 
This would add another poaalbla "cauaa" ahould tha TU group prove to out«> 
perform- the comparlaon group. Thla daalgn would be aa followat 





Beginning 






Group 


of year 


During year 


End of year 


Bxperlmental 


Pre teat 


TU treatment 


Poatteat 


Comparlaon 


Pre teat 


Regular treatment 


Poatteat 



Keep In mind that the above dealgna ahould be uaed vlthln eaoh grade 
level. That lai ualng a TU treatment group from Crade 3 and a comparlaon 
group from Grade 2 or vlce-veraa would not be appropriate. Care ahould 
also be taken to avoid ualng apedal claiaea (e.g., EMR or gifted) aa 
either TU treatttant or comparlaon daaaea If they are not uaad for both. 



If the above guldellnea are followed, a comparlaon of the TU treat- 
ment claaaaa and the comparlaon' daaaea ahould provide a definitive taat 
of the program objecta and thua determine the effectlveneai of the TU 
process in that achool. 

If a roughly comparable comparlaon gvoup la not available, both a 
pretest and a poatteat muat be uaed with the alngle TU experimental group. 
The dealgn would be aa follows i ' 



Pretett at Poitttit at 

beginning TU treaCMnfc and of 

of year ytar ■ ■ / 

This design can not be uaad to eatabliah that the TU traatnant eauaad Cha 
. change but the changa' can be aaaasaad. The Joint Diaaanination Ravif 
Panel will not racogniaa thia deaign aa evidence of efficacy. 

Teeting and Inetrimentation 
\ major component of the Validation proceaa for the TU program tt 
the testing component. The instruments that a«f uised and the tine at 
which the instruments are adminlatered are important to the validity of 
the entire evaluation process.* 

, A number of instruments have beeii^ field 'tested with TU progcama. 
These include the TU program developed Crl^ter^iott-Referanced Teats (Cl^^a) 
(at the preaent time there are 10 taata* one each, for the decialon-makliigt 
forecaating, and planning taleftta{ three for the comaunlcaci^cih talantt 
and four for the productive thinking talent) » Torraflce Teaits of 
Creativity (T(ir ranee, 1966), and vtriq^s atandardiaed achieveiient 

batteriea (auch aa the Stanford A^i^emant Te^, Glymar Barrett, 

■ *' - 
California AchieveMnt Teat, etc.). The lU CHi'a are available from the 

TU of flea in Mobile. The manual indudea copiea of the inatruaants, 

reliability and validity data, and diraotiona ; f or- toorlng them. The 

other teata ate av&ilable c<Mirclall/. 

th^ problem of whan to give the teata la alao Important. Obviously, 

they should be given near the end of the project y|i,ar to^both the TU 

■ • .-At . 

-'I 

treatmeht group and the 'comparlaon group. If atudenta in both groups were 
selected in ooaplataly random Mnnar Indlcatad proviously, than th«t 



is Che only tinit it it ntjcoiiary to glvt tht toiti. Howtvtr, if any 
deviation froa tht random aaiignmant of ttudanti to tvtatpwnt groups 
occurred, the tests must be given both before and after treatment has 
occurred. This means that both the TU treatment group and the eomparismi 
BTdup tnuRt be tested at the beginning of the year and near the end of thm 
year on the talents which vera being taught. This includes all cases 
whore Intact das^iss were used as TU treatment and comparison groups. 
It iB very important that the pretests be given before the students 
receive any instruction In a particular talent. 

Thuf, a variety of instruments are available for use.' A set of 10 
Criterion RAferenctid Talent Tests have been developed spodflcaily to 
Measure the five TU taleata. They should be given both at the beginning 
and the end of the school year unless it was possible to assign studamts 
to groups in a completely random fashion. 

Analysis 

Generally, the analyses can only be done by an individual trained 
In statistics. This sec^on is designed to provide giildancs to such a 
person should he or she be available locally. The TU staff rm^uests 
that regardless of who does the analyses, the TU office be provided with 

It* 

a copy of the original data reporteti oy individual student. Reporting 
forms for this nurpose are located in Appendix B. Reproduce them as 
needed or request multiple copies from the TU ol'flce in Mobile. 
Additionally, information on how the Reject was implemented is needed in 
order to use the information in a combined evaluation of the TU program. 
A Procedures Questionnaire is provided in Appendix C for that purpose. 
Please complete this questionnaire and return it to the TU office along 
with the individual student data. 



rosttcat Only Gaif 

In tht eatt whtrt only a poitttit It glvtn, th* tvp groupi nay bt 
coroparad liilng an Indaptiidant groupa t-taat (Perguaon, 1976, pp. 164-1^6) • 
Thla flrat taqulraa teaclng (ha aquallty of tha population varlancaa 
(Fcrguaon, 1976, pp» 177-1^0). Should they prove aignlflcantly dl£fa|rantt 
.it la reconnended that the Wllcoxon procedure be uaed (Ferguaon, 1976, . 
pp. 3R7-390). The uae of one of the above tf.at> ?nn detenilne If a 
atatletlcally algnlf leant difference exlata bee^e n the two groupa and the 
direction of that difference. ^, 

Pretest-Poatteat Caae 

. ■ if / ■ f 

The najorlty of evaluatlona will require that both the TU treatnant 

group and the comparlaon group receive a preteat and a po^^ttaat. The 

analysla of thla caae la aonevhat more complicated* pattlcitlavly If thi 

groups had preexlatlng dlffarencea. In thla caae an analyala of varlaaot 

(ANOVA) procedure ualng a complatisly randomised model with repeated 

nieasurea (Kirk, 1968, pp. 2A5-284j Llndqulat, 1933, pp. 267-272) la 

recommended. If the two groupa gained differentially, a ilgnlflcant^V 

group by teat Interaction will be obaerved. Thla can be foXloweu ti> 

ualng t-teata to detemlne i^lch group outperformed the other. 

If the raw data ara lant eo tha TU of flea, tha TU avaluatora will 
perform the analyaea for you. If local aaalatance la available, the TU 
of f Ico atlll needa the rav atttdent data plua the completed Proofdurea v 
QuQstlonnalre. In thla caae, the data can be analysed ualng the 
procedurea outlined above. 
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PyoCtdurei Summary 

i 

By using the armpit objeetlvet, inplemantlng the design Indicated, 
administer Irg approprUte tests, and analysing thf data in t^e mahn^r 
Indicated, the efficacy of the TU program In your school can be 
t^atabUshcd. Technical assistance In this regard Is available from the 
'tu office In Mobile. 



jERIC 
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CHAPTER. Ill 
RKPORTXNC RESULTS • . 

t 

A clear and concla* reporting of the reaulte la needed to . * ' 
coiOTunlcate your proJect»8 effectlveneia* Thla chapter provldaa aosie • , ; 
hinta on how to do thla. • v 

Two baa Ic types of Information ara needed to jmalyse a project's V 
•ffectlveneaa. Thaie are Inferential reaulta which Indicate the ilkeUtloo4 
that resulta are alao true In the whole population and deacrlptlve 
reaulta which provide evidence of thi project's iuccoaa with the ssttpli. 
Both typea of reaulta may be reported ualng tablea and/or flgurea. Tht • 
badic raw data on each Individual, atudent will 'not be part of :the prillt^ ' 
rapott but ahould be submitted to the TU office In Mobile as they are 
toinblntd with data from other TU programs from around the country for 
furthar analyala. 

Threa Inferential analyai a procedurea were auggeated In Chapter II » 
These vera the f^teat, Wlleoxon, and the analyala of varlsnct proeaduts. 
The t-teat and wilcMcon procedurea were auggeated for the poatt^st only 
dealgn while the Snalyals of variance was suggeated for the pratsst* 
posttSHt design. The reporting of the raaulta from each procedure will 
ba ndted .acparstely. 



Posttast Only Dssl^yi 



Both the Inferential and the deacrlptlve Information In^iiis f d^^^^ 
only situation can bs rsportsd In ons table* An ixsmpi s- ItT gftjjtt^^^n^ 
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T«blt 1 



Sunnary of Bxptrlmontal «nd ConpArison Dat« 
for Product lv« Thinking 
Or«dos 1-3 



Pltxlbility Originality 



Group Group 



Gradg 




Exp. 


Comp. 




Fvn 


wunp • 


t-valiit 




Mean 


16.8 


7.8 






!IOa9 




1 


*SD 


5.2 


4.1 


7.59* 


14.0 


14.7 


1.15 




N 


.46 


.26 




46 


. 2^ 




» 

2 


Mean 

SD 


19.3 
5.9 


10.4 
6.7 


4.34* 


34. 1 
12.1 


37,3 
24.9 


- ^46- 






14 


35 




14 


35 . 


i" 




Mean 


14.7 


i2.6 




36.9 


24.8 


9 


3 


8D 


3.3 


4.7 


1.31 


19.2 


11.7 


3. a* 




N 


10 


31 




32 


37 





















*KxperiiMntal group aignificantly outparformad tha edmpariaon group 
at tha .05 laval. , 

• ■ 

Nota that for aach gi^oup, tha maan, atandard daviation (8D), and* 
aamplt aita (N) are provided along with the t-value. Thua, an Individual 

x"* .... 

can gluan both the infef^iMitlal and deaeripti* information from the table. 
/ tablt! ahould be able to atand alone, tha. , be interepretad without 
having to refer to tha narrative for further explanation. Thua, tha title 
of the table daa^ribaa what il in the table, A figure or graph .could alao. 
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bo usti to dlspUy th« data but It would not b« any mora Informatlva 
In this eaaa. 

If the variancaa of tha two groupa are net homoganaoua and the 
Wllcoxon procedure is uaed, a aimilar table can be used replacing the 
mean, SD, and t-value with the median, Interquartile range, and Wllcoxon 
U-statistlc reapectively. 

» 

Preteat-Poatteat Dealun 

An analyala of variance procedure la recommended when a preteat- 
posttest deaign ia used. Deacribing the reaulta irequirea at leaat two 
tjblea— a standard analyala of variance aummary table and a table of 
means and atandard deviationa. 

The analysis of variance sunnary table for a oneway. deaign with 
repeated measures is shown below. , 

Table. 2 

Analyala of Variance Summary, Table Comparing 
TU and Compariaon Groupa on Fluency Variable 



Source 


ss 


DF 


' MS 


F 


Between Groupa 


170.1 


1 


170-. 1 


1.72 


Between Error 


9713.9 


95 


98.7 * 

* 




Repeated Measures 


267.6 


1 


267.6 


22. 


Groups by Measures 


233 ;i 


2 


116.6 


9.77* 


Within Error 


1132.87 


95 


11.9 





*Slgnif leant at .01 level. 
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Tho dtticrlptlve information can bt shown in a tablt luch as 

iTablc :}. 
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Table 3 

Descriptive Information on fluency Variable 



G^-oup 



T*J ^Apt rimental 

Conipcit} son 



N 



47 

42 



Pretest 



Mean 



SD 



26.3 
27.1 



7.51 
7.48 



Postteat 



Mean 



SO 



33.4 
27.8 



7.44 
7.49 



Noco lUai Table 3 can be expanded to include multiple gradea, 

multipU? CO np orison variables, etc. . * ^ ' » . 

* • . * 

Tlip same information can b* displayed very effectively in a figure 

(Kraph). ' 



34 
32 

Fluency 30 
28 

r 

26 



i 




TU Group 



Comparison 
Group 



Pretest Posttest 
Figure 1. TU and comparison groups on variable, fluency. 
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Noti? that tht tntlr* "fluency" icale !■ not shown and a c^tout li 

provided to communicate this. This type of figure Is especially 

dramatic when the TU group outperforms the comparison grdup. 
•. 

Reporting Ro s ultB Sutmnnry 
Tlie key luaue when reporting results Is to provide the Information 
needed for someone to Judge the program In as short and< concise a manner 
as posttlble. The Illustrations In this chapter are Intended only as 
guides. Modifications will be needed in many cases but the same Ideas 
can be used* 
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CHAPTjR ZV 
8UMMARX 



Tlie material in thla gulda la Intandad to aaaltt projact avaluatora 
at the achool leval In evaluating TU programa. Obvloualy, It cannot 
answer all of the queatlona which might arlae. Should you encounter a 
problem not covered In thla ^ulde, technical aaalatance la available from 
aevera} aourceat 

Technical Directors Ma. Sara Walrop 

Arlington School 

1107 Arlington Street ^ ' 
Mobile, Alabama, 36606 
(205) 690-8055 

Evaluatorai Dr. Jamea B. McLean 

P. 0. Box 4006"" 
The Unlverilty of Alabama 
Unlveraity, Alabama 35486 
X205) 348-7575 

f Dr. Brad S. Chlaaon 

P. 0. Box Q 

The Unlveralty of Alabama 
Unlveralty, AlabapM 3S486 . 
(203) 348-7575 - 

*■ 

* r 

Dr. John M. Lane 
College of B4ucation 
Unlveralty of South Alabama 
Mobile, Alaba.i)a 36608 
(205) 460-7141 

An Important point In regard to the evaluatifon la that It be planned 
klloia ^ inatructlon begins. Zt, la very difficult and many tlmea 
Impossible to design an effective evaluation after the fact. Poc 
exomplc, preteat results cannot be obtained after atudantS have been 
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! iP'i f - w i ^'ii 1 null \\^^9\Hyf\nwm\m^m^j^i^i^^^ 



txpoacd to the TU proceit. Thui, gome preplanning can' •llmlnate 

» 

evaluation problems later. 

■I 

I 




m 
■1 

'■■4 



■4 

1 

i 

i 



'3 
'3 



4.1 



♦ i 



LERIC 
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APPENDIX A 
SAMPLE TU PROGRAM OBJECTIVES* 



I -^x ample 1 

By the end of the ichool year, students receiving TU 
Instruction In the Productive Thinking talent wlU outperform 
significantly (at the .05 level) students not receiving TU 
Instruction In this talent |is Indicated by the Productive 
Thinking Criterion Referenced Talent Tests. 

* . 

Thlf objective Is Intended for the poittest only designs. It can 
bo modified for any talent by substituting Forecasting, Decision 
Making, Planning, or Conmunlcatlons for Productive Thinking. In addition, 
<>tiier testa^iiuch as the Torrance Tests of Creativity aay be substituted 
for the TU Criterion Referencad Talent Tests'^where appropriate or tised In .. * 
conjunction with the CRTs. , 

If a pretest-posttest design is used, the objectives need to be 
modified slightly to reflect this. Example 2 provides an illustration of 
this for the Dadsion Making. Talent. 



Example 2 ^ ' 

Between the beginning and the end of the school year, 
students receiving TU instruction in the Decision Making 
Talent will outgain students not receiving TU instruction 
significantly (at the .05 level) as indicated by the 
Decision Making Criterion Referenced Test. 

^ « 

This objective takes into account the idea of the TU students 
outgalnlng the others. Both a pretest and post test are needed to 
establish if this does indead happen. 

Sometimes it is desirable to include in objective concerning 
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acadomlc growth. Thii uiu«lly will involve one or mort of tht three baile 
skills— roedltog, writing, end arithmetic. Objectivee for theee cen be 
done eoparetely for erch, much in the aame manner as those for the talent 
objectives or combined into a single academic object. A pbsttest only 
plngU academic objective is illuotrated below. 

Kxampla 3 

By the end of the school year, students receiving TU 
instruction will outperform significantly (at the .05 level) 
in the basic academic skills students not receiving TU 
instruction as indicated by thw California Test of Basic 
Sklllsi 

This objective may employ any standardised test battery. Many 
times It can be tested usinj^he results of a state-wide testing program, 
thus eliminating the need for additional testing. . In the case of non- 
random sampled groupa, the pr^ test-pott test design should be used and 

r 

thus, m objective similar to that Illustrated in Example 2. 



TALENTS UMLIMZTED TEST DATA REPORT ' 
FORM A (TU CRITERION REFERENCED TESTS) 

. Projtct Director • ■ 

City ' ; Seatt ^ ' . . ' 



Student's Name 
(Ootlonal) 


1 Grade 1 


Check 
one** 


rrodu( 


etivt 


roreci 




D«ei 


ii.on 
Ini 


• 


Cominleatlofi 




1 Contro^ 


Pre 


Pott 


Prt 


V 

Post 


V }* 

Pra 


( )* 
Pott 


( )* 
Pra 


< )* 
Pott 


< )* 
Pra 


( )• 

Pott 


















































* 




























































\ 

* 
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AIntsrt daiQ taa^ was admlnlatarad In paranthetci* 

**"TU" inHicatas ttudant raeaivad Talan^t Unllnltad treatment, "Conttol" Indlctttt tha ttudtnt did not. 



TALENTS UNLIMITED TEST DATA REPORT 
FORM B (ACADEMIC AND OTHER TESTS) 

School • Projtct Dltector 

City ■ ^ • ^ Stat* ■ ■ 



Nm of ttit/Subttit 












font/Ltvol 



















■ 




8tud«nt*« Naao 

(Optional) 


Grade 




ick 
1** 


( )* 

Pra * 


( )* 
Poat 


( )* 
Pra 


* )* 
Poat 


( )* 
Pra 


( )* 
Poat 


( )* 
Pra 


( )* 

Pott 


Prt 


( )• 

Peat . 




ConLrel 






































i 






















* 
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• • 
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*Inaart data teat waa adminlaterad* 0 ^ 

^"TU" Indlcatea atudent received Talenta Unllniltad tra#tnant, "ControV* indieataa tha atudant did tioti 
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PROCEDURES QUESTIONNAIRE 



lima o' a. top ting iichool 



(Ch^-ck one) Elementary Middle or Junior High ^ High Schodl 

Name of person filling out thle questionnaire 

' lip 

Address • ■ ' 

. Strtet '. gltJT sutt Zip Codt ' ^ 

•School Prlnfelpsl * ' 



TU Contact Person 



I. Is /our school (check one) 

Public 

Private 

Other 



2. How would you classify your connunlty? (check one) 
• Large city (over 250,000) 

Medium city (100,000 to 250,000) : 

: Small city (lets than 100.000) 

Suburban (within 25 miles of city above) 

Rural 

" ■■ ■ f 

3. Estimate the proportion of TU children which come from homes where the 
breadwinner it i 

Professional/Technical 

Other white collar (e.g., Clerk, Clerical, etc.) 
Blue Qollar tkill labor, ete. 
UntkilUd labor 



. Other . 

?9 
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How wert tht TU itudtott ttlectud? 

Uandomly by ntudont 

Groupt watt aeltcted randomly 

It ■ 4\ 

Croupa vera aaalgned > 

Othar (Pleaae fxplaln) ■ ' 

iiow vara control (compariion) atudanti aalectad? 

^ ^ No control group ' ' 
Randomly by atudatit 

, Groupa vera aalactad randomly < 

: Npn-TU feroupa vara aaalgned aa control 

Othar (Plaaaa axplaln) i " 



Wliai: wora tha taatlng dataa by month and year? 
' Prataat (If any) 



